[V1][Core] Add a cache hit threshold for requests

Purpose

Introduce an optional KV-cache hit-rate gating mechanism, discussed in RFC #24256, to skip requests that are unlikely to benefit from prefill in P/D disaggregated deployments.

Edit: an additional useful scenario for this capability is Request Preemption in P/D disaggregated deployments on a Decode instance. The scenario manifested in llm-d PD tests and involved PD requests that get preempted on the Decode instance: today vLLM simply evacuates such requests KVCache blocks and later retries the requests from scratch. This means the full Prefill work is done internally inside the Decode instance, including all new (possibly many) Output tokens. Tests in the field showed this case leads to Decoders starting to execute prefills and eventually lock up. The main problem is that the external router (such as llm-d / Dynamo / Production Stack) orchestrating PD has no control over this vLLM behavior once the Decode instance received the request. Setting a small cache hit-rate threshold on the request (say 0.001), will reject this Prefill work in case of preemption, and the request will be sent back to the calling Router / Side Car / Worker.

What this PR adds

• Global setting: --global-cache-hit-threshold ([0.0–1.0], default 0.0)
• Per-request override: cache_hit_threshold ([0.0–1.0]) in incoming request ChatCompletionRequest / CompletionRequest (validated in the protocol layer).
• Finish reason: New enum value and string "cache_threshold" exposed via v1 engine API. Requests rejected by this gating return HTTP 200 with finish_reason="cache_threshold" and no output tokens.
• Config visibility & hashing: Threshold is included in VllmConfig and SchedulerConfig.
• Bounds & validation: All threshold values validated to range [0.0, 1.0].

Why

• Enables Decode-first optimization in P/D disaggregation: when computed-token ratio (local+external) over prompt length is below the threshold, we avoid scheduling low-benefit prefills on decode nodes. This reduces wasted work and remote KV transfers when cache reuse is insufficient.

Backwards compatibility

• Default is 0.0 → feature is disabled by default. No behavior change unless the threshold is set globally or per request.

Test Plan

1) Unit Tests

Unit tests check the scheduler logic, including

• request threshold overrides global threshold
• cache hits from local or external KV cache, or both

2) E2E manual tests

Run vllm serve with --global-cache-hit-threshold 0.8 argument to set a some default value. We'll override it in most requests.

vllm serve <model_path> --served-model "Llama-3.1-8B-Instruct" --global-cache-hit-threshold 0.8

Scheduler computes hit_ratio = computed_tokens / prompt_tokens

We will send 4 requests. Note the order of sending them matters as the first request fills the cache other depend on

• First request with cache_hit_threshold: 0 so it’s guaranteed to execute and populate the KV-cache

1. Request1: Short 26 tokens will be the prefix of future requests.

• Following requests are sent with cache_hit_threshold: 0.33

2. Request2: Long prompt ≈ 58 tokens → ratio 16/58 ≈ 0.28 → rejected as ratio below threshold
3. Request3: Medium prompt ≈ 40 tokens → ratio 16/40 ≈ 0.4 → normal generation

• The next request is sent without a cache_hit_threshold field, which means global value of 0.8 will take effect

4. Request4: Medium prompt ≈ 39 tokens → ratio 16/39 ≈ 0.41 → rejected as ratio below global threshold

Request 1) Warm the cache

This run uses cache_hit_threshold: 0 so it’s guaranteed to execute and populate the KV-cache for the base segment.

curl http://localhost:8000/v1/completions   -H "Content-Type: application/json"   -d '{
    "model": "Llama-3.1-8B-Instruct",
    "prompt": "This is the beginning of a long prompt with many tokens, we need a min of 16 to fill the default block size",
    "max_tokens": 20,
    "cache_hit_threshold": 0
  }'

Request 2) MISS case

curl http://localhost:8000/v1/completions   -H "Content-Type: application/json"   -d '{
    "model": "Llama-3.1-8B-Instruct",
    "prompt": "This is the beginning of a long prompt with many tokens, we need a min of 16 to fill the default block size. Then we continue with many words so that the token length will exceed 16*3 and cache hit rate will be too low to pass the test case threshold",
    "max_tokens": 20,
    "cache_hit_threshold": 0.33
  }'

Expected: HTTP 200 with "finish_reason": "cache_threshold"

Request 3) HIT case

curl http://localhost:8000/v1/completions   -H "Content-Type: application/json"   -d '{
    "model": "Llama-3.1-8B-Instruct",
    "prompt": "This is the beginning of a long prompt with many tokens, we need a min of 16 to be the shared prefix but continue with with whatever text tokens we like and keep it medium after all",
    "max_tokens": 20,
    "cache_hit_threshold": 0.33
  }'

Expected: normal generation ("finish_reason" is not "cache_threshold").

Request 4) MISS case using global threshold

Use global threshold set to 0.8

curl http://localhost:8000/v1/completions   -H "Content-Type: application/json"   -d '{
    "model": "Llama-3.1-8B-Instruct",
    "prompt": "This is the beginning of a long prompt with many tokens, we need a min of 16 to be the shared prefix and now continue with different text so the hit rate will be too low",
    "max_tokens": 20
  }'

Expected: HTTP 200 with "finish_reason": "cache_threshold"

Notes

• Exact token counts can vary slightly by tokenizer/model; we go the numbers above using Llama-3.1-8B-Instruct

Test Result

E2E Local smoke tests on a single node:

• Below threshold: responses returned 200 with finish_reason: "cache_threshold" and empty outputs.
  • Validated with debug logs
  • Request threshold:
    • Request cmpl-410004b615a54d73b7e9f0deebf2b852-0 rejected: cache hit rate 0.28 < threshold 0.33 (request)
  • Global threshold:
    • Request cmpl-6d66ba796f9247fcadca54ae428bf790-0 rejected: cache hit rate 0.41 < threshold 0.80 (global)
• At/above threshold: normal token generation.
• Validators rejected out-of-range values and accepted on boundaries 0.0 and 1.0 (not detailed above)